Skip to content

fix: retry one-shot Thrift CloudFetch 401/403/404 downloads and time out stalled attempts - #610

Open
vuanhphung wants to merge 10 commits into
mainfrom
fix/cloudfetch-download-recovery
Open

vuanhphung wants to merge 10 commits into
mainfrom
fix/cloudfetch-download-recovery

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

On the Thrift backend, a one-shot 401, 403 or 404 from cloud storage on a CloudFetch download failed the query with CloudFetch HTTP error 403 Forbidden, because HttpRetryPolicy retries only 429 and 5xx. A stalled download attempt was never timed out, so it waited until retriesTimeout (15 minutes by default) instead of being retried. Found by databricks-driver-test CLOUDFETCH-001/002/003/007, live against a warehouse with proxy-injected failures.

The CloudFetch spec was relaxed for this. For 001/002/007, a one-shot 401/403/404 must not fail the query, and a link refresh or a same-URL retry both count; JDBC and the kernel already recover with a same-URL retry. For 006/008/009, a 400/405/412 may end in a clear terminal error naming the status, which Node already does; Go doesn't retry them, and Python's Thrift path doesn't retry a 400. So this PR no longer refreshes links or retries other 4xx.

The fix:

  • 401/403/404: the download marks the response retryable, and HttpRetryPolicy retries the same URL within its existing attempt budget, backoff and the per-download retriesTimeout deadline ([coverage] Conformance findings: CLOUDFETCH-018 #520). When attempts run out it fails with RetryError (… (last error: HTTP 403: Forbidden)).
  • Stalls: socketTimeout is now also the inactivity timeout of each download attempt, restarted when headers and each body chunk arrive, and a timed-out attempt is retried as ETIMEDOUT. It takes effect when socketTimeout is shorter than retriesTimeout. connect({ socketTimeout }) now reaches the download path.
  • 400/405/412: unchanged; the download fails at once with CloudFetch HTTP error <status>.
Spec (Thrift) Before After
CLOUDFETCH-001 / 002 / 007 (401 / 403 / 404) 🐞 query fails ✅ same-URL retry
CLOUDFETCH-003 (stalled download) 🐞 waited out ✅ timed out after socketTimeout, same-URL retry
CLOUDFETCH-006 / 008 / 009 (400 / 405 / 412) fails with CloudFetch HTTP error <status> unchanged (meets the relaxed contract)

SEA/kernel paths are unchanged. Size: +46/−7 non-test lines (it was +277/−75 before the spec was relaxed).

Testing:

  • Unit tests against a local HTTP server with the real HttpRetryPolicy: a one-shot 401/403/404 is retried with the same URL; a persistent 403 ends in RetryError naming HTTP 403; 400/405/412 fail after one GET; a stalled attempt is timed out and retried, with no URL in the error; a body that keeps arriving is not cut off. Also the policy's retryable flag and connect({ socketTimeout }). Full unit suite: 1784 passing; lint, type-check and prettier pass.
  • Live (driver at this head, kernel 0b5d0ed, driver-test main e40743249, whole Thrift CloudFetch spec): 003 turns into XPASS. For 001/002/007 the query succeeds (143,672 rows) after a same-URL retry; driver-test main still asserts the old refresh contract there, and its tripwires are being rewritten to the relaxed one. 006/008/009 fail with CloudFetch HTTP error 400/405/412 as the relaxed contract allows, and 004/005/010..017 pass.

Follow-ups (not in this PR):

  • Refresh a link that is already past expiryTime instead of failing with CloudFetch link has expired.
  • Read and discard non-OK response bodies before a retry.
  • Document the CloudFetch role of socketTimeout in CONNECTION_PARAMETERS.md.

This pull request and its description were written by Isaac.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly solid, with 2 concerns about timeout semantics. The per-attempt socketTimeout measures total elapsed time, while on Thrift requests it is an inactivity timeout, so lowering it can now fail large or slow downloads. Separately, link refreshes start a fresh retriesTimeout deadline each time, which loosens the end-to-end bound added in #520.

Comment thread lib/result/CloudFetchResultHandler.ts Resolved
Comment thread lib/result/CloudFetchResultHandler.ts Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

vuanhphung and others added 3 commits November 15, 2026 07:12
…stalls

A one-shot 401, 403 or 404 from cloud storage failed the query: CloudFetchResultHandler threw on any non-OK response and HttpRetryPolicy retries only 429 and 5xx. Other 4xx responses failed the same way, and a stalled attempt was never timed out, so it waited until retriesTimeout instead of being retried.

On 401/403/404 the presigned URL itself was rejected, so fetch a fresh link for the same rows with FetchResults(startRowOffset) and download it, counting each refresh against retryMaxAttempts. RowSetProvider serializes its FetchResults calls and pages by explicit offset after a refresh, since the refresh may move the server cursor. Retry other 4xx with the same URL, and apply socketTimeout to each download attempt so a stalled one is retried.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
… refreshes

Review follow-up. The per-attempt download timer now restarts on every received chunk, so socketTimeout aborts only an attempt that stops receiving data, not a large download still in progress. Link refreshes reuse the download's retriesTimeout deadline instead of starting a new one each.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
… the trickle test margin

The CHANGELOG now says the idle retry applies when socketTimeout is shorter than retriesTimeout (both default to 15 minutes). CONNECTION_PARAMETERS.md notes that socketTimeout also bounds each Thrift CloudFetch download attempt and that link refreshes count against retryMaxAttempts. The trickling-body unit test had only 100 ms of slack per gap; it now uses a 300 ms idle timeout over a ~1 s body.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

…ix refresh-budget and socketTimeout docs

Every URL in the shared-deadline test now answers 403 after 120 ms, so a fresh retriesTimeout per refreshed URL would refresh 9 times and fail with the 403, while the shared deadline times out within 3 GETs. The docs now say a download tries at most retryMaxAttempts URLs, each retried under its own budget, and the socketTimeout JSDoc gives the Thrift and kernel semantics and defaults. fetch() takes deadlineAt explicitly instead of defaulting it.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low-severity test gap. I traced the refresh loop, the shared deadline, the per-attempt idle abort (including which abort sets timeoutError and which sets ETIMEDOUT), and the offset tracking for prefetched and direct results in RowSetProvider, and found no correctness problems. If refreshLinks itself fails, the caller sees that Thrift error instead of the original CloudFetch 403; this is acceptable, since the query failed before this change too.

Comment thread lib/DBSQLClient.ts Resolved
The CloudFetch download idle timeout reads socketTimeout from ClientConfig, so the existing connect() retry-option tests now also cover socketTimeout: a set value, 0, and the default when omitted.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Nit

Looks good. The refresh, same-URL retry and idle-timeout logic matches the description, fetchNext and refreshLinks are serialized correctly, and the new tests cover the main paths. One low edge case: the loop does not check the shared deadline before refreshing a link. Follow-ups, not caused by this PR: links already known to be expired still fail upfront with CloudFetch link has expired instead of being refreshed, and non-OK response bodies are still left unconsumed across retries.

Other findings

  • ⚪ Nit — The refresh loop does not check whether the shared retriesTimeout deadline has already passed before it calls refreshLink. Suppose the deadline runs out while the 403 GET is in flight, or during the FetchResults refresh, which the deadline does not bound. The loop still issues the refresh RPC and then calls fetch with deadlineMs = 0. That fetch starts a GET, aborts it on the next tick, and throws RetryError(TimeoutExceeded, undefined). That call's lastFailure is still unset, so the message is a bare Retry timeout exceeded. The 403 that actually caused the failure is lost, and you pay for one extra FetchResults round trip and one doomed request.

Suggested fix: in the loop condition, stop when deadlineAt !== undefined && Date.now() >= deadlineAt. The loop then falls through to the existing CloudFetch HTTP error 403 … path.

…sTimeout

The refresh FetchResults RPC is not bounded by the download deadline. If it ran past it, the next fetch started a GET that aborted at once and failed with a bare 'Retry timeout exceeded'. Now the download fails with the same TIMEOUT_EXCEEDED RetryError, with the rejected 401/403/404 response as its cause, and without the doomed GET.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Re the review-body Nit on 0411453 (refresh loop and the shared deadline): fixed in 723056c. The meaningful window is the refresh FetchResults RPC, which the deadline does not bound; a 401/403/404 that arrives before the deadline has already cleared the GET's own deadline timer. So the check sits right after the refresh: if the deadline has passed, the download fails with the documented TIMEOUT_EXCEEDED RetryError (kept for consistency with #520 rather than the plain CloudFetch HTTP error 403), with the rejected response as its cause, and no doomed GET is started. New unit test: the refresh takes 400 ms with retriesTimeout: 300, so the error is CloudFetch result download failed: Retry timeout exceeded (last error: HTTP 403: Forbidden) after one GET. Without the check it fails with the bare Retry timeout exceeded. The other two items you listed (refreshing links already known to be expired, unconsumed non-OK bodies) predate this PR and are left as follow-ups.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

The retriesTimeout deadline was a timer inside each GET, cleared before a
link refresh, so a refresh that was queued behind another FetchResults call,
or whose RPC was slow, kept the download pending past the deadline (forever,
if it never returned). One deadline now spans the whole download, refreshes
included; a refresh that returns late can't resume the download, and a
refresh still queued when its download gives up is skipped.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good: one low test-coverage gap. I traced the refresh loop and deadline races, the idle-timer cleanup, RowSetProvider serialization, and paging by offset after a refresh, and found no correctness issues. As a nit, the new CHANGELOG entry has no PR reference (#610), although the neighbouring entries have theirs.

Comment thread lib/result/CloudFetchResultHandler.ts Resolved
A refresh response without a link at the rejected link's row offset (only links for other rows, or none) must fail the download with the original HTTP error after one refresh, and must never download another chunk's link in its place. Also adds the PR number to the CHANGELOG entry.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung

Copy link
Copy Markdown
Collaborator Author

Re the review-body nit on 3a34b1f: the CHANGELOG entry now ends with (#610) (f47d1a6).

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

…need

Drop the hardening that no CLOUDFETCH cell or regression guard needs: skipping a queued link refresh after its download gave up (its late result is already ignored), two deadline checks that the per-attempt check already covers, and the socketTimeout/retryMaxAttempts notes in CONNECTION_PARAMETERS.md and the ConnectionOptions JSDoc. DownloadDeadline.race is now a plain Promise.race, fetch() takes init as optional again so PlainHttpTransport.test.ts is unchanged, and the CHANGELOG entry is shorter. Tests that only covered the removed parts are dropped.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with 1 low concern: a refreshed link is matched only by its start row offset. I checked the deadline, abort, serialization and offset-paging logic against the callers (ThriftOperationBackend is the only one that builds CloudFetchResultHandler/RowSetProvider) and found no correctness problems. The follow-ups the PR lists (draining non-OK bodies, refreshing an expired link) are pre-existing gaps, not regressions.

Comment thread lib/result/CloudFetchResultHandler.ts Resolved
…d of refreshing the link

Per the relaxed CloudFetch spec (a one-shot 401/403/404 must not fail the query; a refresh or a same-URL retry both count, and other 4xx may fail with a clear error), drop the link refresh, the RowSetProvider offset paging and serialization, the cross-refresh download deadline and the other-4xx retry opt-in. A CloudFetch download now marks a 401/403/404 response retryable, so HttpRetryPolicy retries the same URL within its existing attempt budget, backoff and the #520 per-download deadline. 400/405/412 keep failing with 'CloudFetch HTTP error <status>'. The per-attempt socketTimeout idle timeout stays.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung vuanhphung changed the title fix: recover Thrift CloudFetch downloads from rejected URLs, 4xx and stalls fix: retry one-shot Thrift CloudFetch 401/403/404 downloads and time out stalled attempts Nov 16, 2026

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

This branch was successfully deployed

1 active deployment
azure-prod — 812281e6 Deployed Nov 16, 2026 by vuanhphung via e2e-test (22.23.4, 22_23_4) #2063
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant